Skip to content

5.4. Model Gateway

What is the application contract?

The Python agent keeps its OpenAI-compatible provider and changes only the endpoint from direct Ollama to agentgateway:

AGENT_MODEL_PROVIDER=openai-compatible
AGENT_MODEL=qwen3:4b
OPENAI_BASE_URL=http://127.0.0.1:4000/v1
OPENAI_API_KEY=local-ollama

The marker key satisfies the client library in the default unauthenticated host profile. It is not an Ollama credential. The Kubernetes profiles and the opt-in secured host profile can enforce a configured value as a gateway API key (Chapter 5.5).

How is local Qwen3 configured?

backends:
  - ai:
      name: ollama
      hostOverride: localhost:11434
      pathOverride: /v1/chat/completions
      provider:
        openAI:
          model: qwen3:4b

The k3d profile changes hostOverride to host.k3d.internal:11434. The GKE profile instead declares Vertex gemini-3.5-flash plus backendAuth.gcp, which obtains a token from the gateway pod's Workload Identity.

How do you call the gateway directly?

With Ollama and the host gateway running:

curl -fsS http://127.0.0.1:4000/v1/chat/completions \
  -H 'Authorization: Bearer local-ollama' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b",
    "messages": [{"role": "user", "content": "Reply with exactly: gateway ready"}],
    "temperature": 0
  }' | jq -r '.choices[0].message.content'

The response may include model-specific reasoning or formatting; the transport checkpoint is a valid choices[0].message without a direct request to port 11434.

Does one endpoint make every provider identical?

No. Tool-call formats, context limits, JSON behavior, token accounting, content policy, latency, and model quality differ. The gateway stabilizes transport and policy placement; evaluations decide whether a backend is compatible with the Ops Copilot.

What happens when the provider fails?

agentgateway surfaces an upstream failure. ADK's model error callback returns a stable MODEL_UNAVAILABLE response while logging the exception for operators. It does not silently switch models, because untested failover can change safety and cost behavior.

What is the model checkpoint?

Call the gateway directly, then run one read-only A2A request through the agent. On the host, inspect the gateway JSON log/metrics and the application's model span as separate evidence; the host gateway intentionally exports no OTLP trace. Chapter 6's Kubernetes profile adds the gateway span to the shared trace path. Run the recorded eval set before treating a different model name as supported.